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Abstract 

A perennial issue facing survey research is obtaining satisfactory response rates. One 
method for improving rates is to offer multiple modes for response. An associated issue is 
the effect of differing modes. This research examined both matters using a survey of 
community college graduates. Fielding consisted of two phases. Phase One offered three 
modes for response, while Phase Two consisted of telephone interview of non- 
respondents to Phase One. Response rates, measures of data quality, demographic 
variables and attitude variables are compared across modes. The implications for 
institutional research are discussed. 
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Mixed mode effects in a community college graduate survey 

One of the primary issues facing survey research is how to obtain satisfactory 
response rates (Cochran, 1977; Kish, 1965: Groves, Dillman, Eltinge & Little, 2002; 
Mangione, 1998). Survey nonresponse occurs at the unit (failure to obtain any response 
from a survey unit) or item (failure to obtain responses to individual questions) level and 
results in missing data (Dillman, Eltinge, Groves & Little, 2002). This is a problem 
because it introduces less efficient estimates, leads to issues in using statistical 
procedures and may bring in biases due to systematic differences between survey 
respondents and nonrespondents (Huisman & van der Zouwen, 1998). This is becoming 
increasingly a concern, as unit response rates in some situations appear to be declining 
(Atrostic & Burt, 1999; Atrostic, Bates, Burt, Silberstein & Winters, 1999; Cambum, 
Gunther-Mohr, & Lessler, 1999; de Leeuw & de Heer, 2002; Steeh, 1981). Item 
nonresponse is also an issue and has been studied for a number of years (for example: 
Ferber, 1966; Mason, Lesser & Traugott, 2002) as it results in missing data values and an 
overall reduced data base. Employing methods to increase unit response rates must be 
tempered by concerns over the potential for higher item nonresponse. 

It is important for institutional researchers to use survey methods that maintain 
acceptable response rates. The fundamental role of institutional research and planning in 
post-secondary education is to provide the analytic inputs to facilitate decision-making 
(Frost, 1993: McLaughin, Howard & McLaughin, 1998). These analytic inputs must be 
founded on sound research methods. The purpose is to improve institutional intelligence 
and augment institutional learning and advancement (McLaughlin & Howard, 2001). 
Institutional research and planning brings knowledge to bear on the future of the 
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institution (Peterson, 1999) and this is often through survey research strategies. In post- 
secondary educational institutions the design, collection, analysis and reporting of survey 
work must adhere to the highest standards of research. This will increase in importance, if 
Keller (1995) is right that senior academic management will be more closely tied to 
quality institutional research in the future. 

A significant volume of work has been undertaken to identify ways to improve 
response rates (see Groves, Cialdini & Couper, 1992; Janota, Baum & Slater; 1999). 
Respondents vary in their preference for survey mode and may appreciate being able to 
choose their own response mode (Groves & Kahn, 1979; Swoboda, Muehlberger, 
Weitkunat & Schneeweiss, 1997). Mixed mode strategies have been found to be 
successful (Schaefer & Dillman, 1998). However sometimes giving respondents a choice 
of response mode does not improve response rate, even though one mode may be 
substituted for another (Dillman, Clark & West, 1995). However sequential changes in 
modes appear to improve response rates (Shettle & Mooney, 1999). These may be 
effective not only because people have mode preferences but also because having 
different modes highlights the importance of the survey and encourages response 
(Dillman, 2000). 

Traditionally, surveys have been conducted by face-to-face interviews, telephone 
interviews or mail questionnaires. There are a number of newer methods of conducting 
surveys, many relying on computer assisted techniques. Modes such as e-mail and web- 
based techniques for surveying have evolved quickly and are becoming useful additions 
and alternatives to traditional modes (Dillman, 2000). Nathan (1998) argued that new 
technological developments in communication imply that mixed mode surveys will 
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become predominant. Newer modes include touch-tone data entry (TDE) where 
respondents reply by a telephone touch-tone pad to recorded instructions, (this is also 
sometimes referred to as interactive voice response (IVR) when the predominant 
response is through verbal recordings), and Internet (Web) based surveys, either through 
e-mail methods or through a URL link to an electronic questionnaire for prospective 
respondents. Borden, Massa and Milam (2001) contend that institutional research will 
expand its use of on-line surveys in the future. Such use must adhere to high survey 
standards, including an understanding of the appropriate protocols and effects of newer 
survey modes. 

Generally, web-based response rates, in comparison with traditional modes, have 
been low (Abraham, Steiger & Sullivan, 1998; Couper, Blair & Triplett, 1997: Sheehan 
& McMillan, 1999). There seems to be little research on response rates with TDE. 

Although employing a mixed mode approach to survey research may result in 
improved unit response rates; there is a concern with whether or not such a strategy 
biases results. There are many aspects to this. Employing a mixture of data collection 
modes in survey research to increase unit response rates may lead to higher item 
nonresponse rates. Generally the respondent mean is a function of two sources - unit and 
item nonresponse. If the frequency of item nonresponse increases with additional 
methods used to increase unit response, there may be a loss in estimates. Hence, overall 
data quality may be affected by a mixed mode strategy. However, there are many factors 
influencing item nonresponse including interviewers, question topics and structures, 
question difficulty, respondent attributes, along with survey mode (Dillman, et. al., 2002). 
The research on mode impact on item nonresponse has been mixed, (see Yun & Trumbo, 
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2000). In some situations open-ended questions have been answered more fully on-line 
compared with paper and pencil (Bachmann, Elfrink & Vazzana, 1996; Schaefer & 
Dillman, 1998). Tourangeau, Rips & Rasinski (2000) found that item nonresponse was 
more likely in self-administered survey questionnaires than those conducted by 
telephone. Stanton (1998) found that a web base survey had fewer missing values than 
paper and pencil. 

If respondents do vary in their preference for survey mode (Groves & Kahn, 
1979), they may vary on key demographic variables. As well, offering different modes 
may affect the representativeness of respondents to the population under study. 

Different survey modes may also lead to mode difference in response, that is, 
respondents in one mode may have responded differently in another mode. Mode 
differences in response have been observed (Dillman, Sangster, Tamai, & Rockwood, 
1996; Schwartz, Strack, Hippier, & Bishop; 1991). This has been attributed to 
differences between self-administered and interviewer controlled questionnaires and the 
difference between aural and visual presentations of questions (Dillman, 2000). In an 
interview situation a respondent interacts with another person who controls the asking of 
questions. Such a circumstance may evoke a social norm approach to responses leading 
to social desirability and acquiescence in response. Schuman and Presser (1981) 
suggested that respondents are more likely to agree than disagree when interacting with 
people. Cognitive differences arise from visual versus aural communication. Tamai and 
Dillman (1992) found such differences in responses to a sequence of satisfaction with 
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The potential for improving response rates by offering multiple modes of survey 
response is examined in this research along with the effects of differing modes. A large 
Western Canadian comprehensive community college has conducted a census survey of 
graduates for several years, using a mixed modes strategy. This involved an initial phase 
of two waves of mail surveys followed by a telephone survey of nonrespondents. The 
strategy recognized the finding that some individuals require prompting through multiple 
contacts to complete surveys (Brennan & Hoek, 1992; Dillman, 2000). College staff 
conducted the first phase while a professional survey research company undertook the 
second phase. Mail was less costly than telephone and the intent was to capture a larger 
portion of graduates through the mail portion. It was observed that over the last four years 
the response rate (calculated as number of completed/number in the census) for Phase 
One declined from 54.3% to 42%, while the rate for Phase Two increased from 13.5% to 
22.5%. Interestingly the first wave response rate declined from 37.9% to 25.6% while 
the second wave rate was stable. 

These findings, along with intent to explore the potential of employing newer 
survey strategies, led to a study of different modes of survey fielding with the graduate 
satisfaction and employment survey. This research had four main objectives. The first 
was to examine the impact of a mixed mode survey strategy on unit response rates. This 
involved an examination of the separate and cumulative unit response rates for different 
response modes offered to respondents. It was hypothesized that offering multiple modes 
of response would increase overall response rates in Phase One of the survey. It was also 
hypothesized that there would be variations in response rates across modes. As well, it 
was thought that offering a mixed mode sequence would increase unit response rates 
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considering both phases. The second objective was to examine the impact of a mixed 
mode survey on item nonresponse. The expectation was that there would be variations in 
item nonresponse by mode with telephone exhibiting less item nonresponse on closed 
ended questions. It was also expected that respondents through the web-based mode 
would provide a higher volume of open comments. Third, this research investigated the 
impact on the representativeness of respondents through a mixed mode survey mode. It 
was expected that respondents would vary on key demographic variables across modes 
but that the aggregate of respondents would closely align with nonrespondents. In 
addition it was hypothesized that offering a sequential mode would improve the 
representativeness. The expectation was that respondents in the sequential phase would 
be more like nonrespondents. The last objective was to examine respondent response 
similarities and differences by mode. Mail and web-based survey modes are visual and 
are controlled by the respondents. The telephone mode is aural and controlled by an 
interviewer. The TDE mode is a mixed presentation. Respondents listened to a taped 
recording asking (not a live person) of questions. Although the presentation was aural, 
they also had a paper copy of the questionnaire to follow, if they wished. For research 
purposes, It was hypothesized that (a) Phase One respondents would not exhibit 
differences in attitudes, due to an attempt to have the survey instrument nearly identical 
across modes, and (b) telephone respondents would be more likely to express agreement 
with attitude items, that is, telephone respondents would have higher mean scores than 
the other three modes. A concern with this aspect of the research is that if the response 
rates vary widely across modes, which is expected, any differences may be due to the fact 
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that people with different views may be more likely to respond by one mode than 
another. 

Methods 

Using a data set of all 1999-2000 graduates from certificate and diploma 
programs at the community college, fielding consisted of two phases for the census 
survey. Phase One was conducted by offering three modes for response, by mail, web- 
based and touch-tone-telephone data entry (TDE). The mail and web-based components 
were administered by college staff while the TDE mode was conducted by an external 
professional survey research company. Phase Two consisted of telephone interview by a 
different external professional survey research company of nonrespondents to Phase One. 

At the community college the Graduate Satisfaction and Employment Survey is 
conducted annually to assess outcomes for graduates and their perceptions of the college 
experience. The survey is also a component of the accountability requirements for the 
college. Results are reported by individual program, division and college wide every year 
in a public document. The total number of 1999/2000 graduates was 1,606, for 123 no 
addresses or phone number could be obtained or the initial mailing was returned as 
moved, etc., leaving a deliverable frame of 1,483 graduates. Male graduates represented 
52.9% of the census. The mean age at graduation was 25.7 ( SD 6.7) and the median age 
was 23.3. Diploma students represented 59.7% and certificate students 40.3% of 
graduates. These variables were derived from institutional data and merged with the 
survey dataset. The number of institutional variables were limited due to issues 
surrounding informed consent and the protection of the privacy of personal data 
legislation. 
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Fielding protocol for the First Phase included a survey questionnaire and 
personalized letter sent to each student who graduated. This initial mailing presented 
graduates with three alternatives for responding: through a stamped, addressed return 
envelope, a URL for a web based survey, and a TDE mode. Prospective respondents were 
supplied with a unique identifying code to access the latter two modes. A second survey 
and letter were sent to those who did not respond by any mode to the initial request. 

Phase Two involved a telephone survey of the non-responding graduates to the First 
Phase. The telephone survey instrument modeled the mail questionnaire and the follow- 
up protocol involved up to five callbacks. 

The survey instrument was relatively short, consisting of a total of sixteen 
questions (not all respondents would answer all questions) with an opportunity for open- 
ended comments. The questions explored aspects of graduates’ employment or 
continuing education circumstances and satisfaction with their college education. The 
web-based survey instrument modeled the mail questionnaire and the TDE voice 
instructions paralleled the mail questionnaire, which all respondents would also have 
been able to follow. Closed questions were answered through keying while open 
questions were answered by voice and recorded. 

The independent variables for this research were mode and phase. The dependent 
variables are described in relation to the research objectives. 

The impact of modes will be evaluated using unit response rates. Rates were 
calculated for each of the modes and for each of the phases. These rates were compared 
with previous years and across modes and phases. Where relevant chi-square was used to 
assess significant differences. 
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Data quality was investigated through an analysis of the care taken in completing 
the survey instruments by reviewing the item completion rates across modes. This also 
was an attempt to investigate the degree of involvement of respondents in the survey. 
Considerations included: missing values across modes for fixed response questions, 
missing values on open response category questions, the number of respondents 
providing comments or suggestions for improvements, and the number of No Opinions 
(the attitude questions had four categories with an additional category to record No 
Opinion). Differences were assessed through chi-square. In additional comment volume 
was compared across modes by mean word counts and assessed through one-way 
analysis of variance. 

Nonresponse error related to key demographic variables was also evaluated. The 
variables were gender, age, monthly earnings, achievement (certificate or diploma) and 
current activity. Respondents by mode in Phase One were compared, respondents across 
all modes and the overall Phase One was compared with Phase Two. Differences by 
gender, current activity (employed/self-employed, student, unemployed) and achievement 
were compared through chi-square. Age and monthly earnings means were compared 
through one-way analysis of variance across modes. In addition, comparisons with 
nonrespondents on gender, achievement and age were calculated to understand the 
representativeness of the response set over modes. 

The response differences on variables assessing attitudes were compared across 
modes. Seven questions on the survey instrument asked respondents to indicate their 
degree of agreement with statements regarding: whether they would recommend their 
program to others, satisfaction with education received, satisfaction with the currency of 
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program, whether instructors were knowledgeable, quality of training materials, 
appropriateness of the technology, and reasonableness of the cost. A one-way analysis of 
variance was conducted for Phase One modes to determine mode effects. A one-way 
analysis of variance was also conducted for all modes to determine if an interviewer 
controlled situation resulted in more respondent agreement. To investigate this more 
fully, the seven items were considered to be a scale and a factor analysis was used to 
confirm this and subsequently the summated scale scores were used to assess attitude 
differences first for Phase One to determine if there were mode effects and then across all 
modes to determine if there were effects related to varying types of modes, such as self- 
administered (Mail and Web-based) combined mode (TDE, although it was primarily 
self-administered) and interviewer controlled (telephone). 

Results 

Response Rates 

The response rates for all modes in the survey of 1999-2000 graduates are 
displayed in Table 1. After the first wave, the response rate was 20.7% and after two 
waves of mail-outs, 35.2%. These rates were unexpectedly lower than the previous four 
years. Table 2 displays the response rates for surveys of graduates from 1995-1996 
through 1999-2000. The first phase census response rate was 42.0% for 1998-1999 
graduates, 41.9% for 1997-1998 graduates, 41.6% for 1996-1997 graduates and 54.3% 
for 1995-1996 graduates. All of these earlier surveys were conducted with postal return 
only for the first phase. 

Phase One respondents were more likely to have cooperated through mail (% 2 (2, 
n = 565) = 398.69, p < .0005). Response rates for the three initial modes were: Mail, 
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25.6%, TDE, 4.5%, Web-based, 5.0%. These response rates varied greatly over modes 
and are similar in general to others reported (Di liman, 2000), although the rates found in 
this research may be even lower. By far mail was the most effective response mode of the 
first phase. The TDE and Web-based modes had very low response rates. Both of these 
modes required respondents to access a secondary means, that is, a touch-tone telephone 
or a computer with Internet access, to respond. This may have acted as a deterrent or 
barrier. 

The response rate for the second phase (telephone) was 29.6%, yielding an overall 
response rate of 64.8%. This compares with an overall response rate of 64.6% for 1998- 
1999 graduates, 60.1% for 1997-1998 graduates, 71% for 1996-1997 graduates, 67.8% 
for 1995-1996 graduates, all conducted with sequential modes of mail only followed by 
telephone. Through all of these surveys the use of a second mode has resulted in 
substantially increased overall response rate, more than could be achieved through 
additional First Phase waves. Mangione (1998) indicated that subsequent waves of mail 
surveys achieve response rates about half of the previous wave. 

Care taken in completing surveys 

An important part of this research was to understand the degree of care taken in 
completing the survey instrument by mode type. As well, this measured the degree of 
involvement in the survey in relation to survey mode, particularly through No Opinions 
and comment volume. The missing values and rates are presented in Table 3. There were 
thirteen closed response category questions. The questions asked about current activity, 
reasons for being unemployed, when a job was accepted, whether work was full or part 
time, whether the job was related to training at the College, how a job was found, and 
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seven questions about graduates’ attitudes towards their experience at the College. The 
overall missing values percent (for eligible respondents for each question) was 0.5%. 

The rates for the modes were, Mail, 0.7%, TDE, 3.1%, Web-based, 0.3% and Telephone, 
0.0%. Respondents through the web were more likely to have missed completing the 
closed ended questions than mail respondents. In the context of a telephone survey, the 
interviewer controls the stimulus and, in this survey, achieved full completion of all the 
closed category questions. The seven attitude question included a No Opinion response 
category. There were no significant differences on the rate of selection of No Opinion 
among modes. There were three opened ended questions asking for salary, position title, 
and employer name. The overall missing value percent was 22.7%. The percents for the 
modes were, Mail, 19.3%, TDE, 32.6%, Web-based, 13.9% and Telephone, 25.4%. 
Respondents through the TDE mode were more likely not to complete these questions 
than respondents through any of the other modes, while telephone respondents were more 
likely to miss questions than web respondents (see Table 4). 

The questionnaire provided respondents with the opportunity at the end to provide 
comments or suggestions for improving program educational quality at the College. 
Overall 52.2% provided comments. The rates for the modes were, Mail, 54.4%, TDE, 
28.4%, Web-based, 54.3% and Telephone, 53.8%. Respondents through the TDE mode 
were more likely not to be involved with the survey than respondents through the other 
three modes (% 2 (3, n = 1041) = 20.622, p < .0005). 

Considering those 543 respondents who provided comments, the mean number of 
words included for all modes was 26.7. The means for the modes were Mail, 37.0, TDE, 
48.9, Web-based, 81.2 and Telephone, 6.7 (see Table 5). A one-way analysis of variance 
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revealed significant differences between modes as was expected. A post hoc comparison 
using Bonferroni revealed that web-based mode respondents provided significantly more 
content than the other modes and that telephone respondents provided significantly less 
content than the other modes. Telephone comment volume was lower likely as an artifact 
of the interview situation where the interviewer would input abbreviated comments as a 
respondent spoke and then read back the comments and not provide a verbatim recording. 
It has been suggested that web-based surveys have higher volumes due to the speed of 
key stroking over handwriting (Bachman, et. al., 1996). 

Survey mode and nonresponse errors related to demographic variables 

There were two methods for comparing nonresponse error related to the 
demographic variables. The first evaluation compared reported demographics across 
modes and across phases. The variables were gender, age, monthly earnings, 
achievement, and current activity. The latter three variables were also outcome variables. 
Table 6 presents the number and percent for gender, current activity and achievement, 
and means and standard deviations for age and monthly earnings by mode and phase. 

There were no significant differences in respondents by current activity or age 
across modes, within Phase One, or from Phase One to Phase Two using chi-square. 

Most respondents were employed/self-employed. A one-way analysis of variance was 
calculated with monthly earnings as the dependent variable and survey mode as the 
independent variable. There was a significant effect of mode, F( 3, 714) = 5.291,/? = .001. 
A post hoc comparison using Bonferroni showed that respondents through the web 
reported significantly higher monthly earnings than other mode respondents. The overall 
mean for all respondents was $2,043 (Mdn=$ 1,920), and for mode respondents at $1,969 
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(Mtfn=$l,872) for Mail, $2,032 (Mdn=$l,976) for TDE, $2,425 (Mdn=$ 2425) for Web- 
based and $2,046 (Mdn=$ 1920) for Telephone. Considering gender, female respondents 
were more likely to be early responders, that is in Phase One (58% out of total female 
respondents) than males (51% out of total male respondents) and this was significant, % 2 
(1, n = 1037) = 5.405, p = .02. Overall there were more male graduates than female. 
Graduates from the College receive either a certificate for a one year program of studies 
or a diploma for a two year program of studies. Respondents who achieved a certificate 
were more likely to have responded in Phase Two (telephone mode) % ( 1 , n = 1041) = 
13.191, p<. 0005). 

The second evaluation compared age, gender and achievement of respondents by 
mode and phase with nonrespondents. This second analysis aimed at understanding the 
representativeness of respondents. As Table 7 illustrates, survey respondents were not 
different from nonrespondents with regard to gender. This was also true for all gender 
comparison of individual mode respondents to nonrespondents: mail and nonrespondents 
(% 2 (1, n = 976) = 1.638, ns), TDE and nonrespondents (% 2 ( 1, n = 637) = .527, ns), Web 
and nonrespondents (% 2 (1, n = 644) = .892, ns), telephone and nonrespondents (% 2 (1, n = 
1040) = 2.095, ns). Table 7 also shows that survey respondents were not different from 
nonrespondents by age. This was also true for all age comparison of individual mode 
respondents to nonrespondents: mail and nonrespondents (t(968) = .692, ns), TDE and 
nonrespondents (t(630) = -.064, ns), Web and nonrespondents (t(638) = -1.361, ns), 
telephone and nonrespondents (t(1031) = -.590, ns). These findings suggested that each 
of the mode’s respondent set was similar to nonrespondents, providing overall comfort in 
the utility of the survey findings. Respondents were different from nonrespondents by 




18 



Mixed Mode 18 



achievement, as graduates with certificates were more likely to be nonrespondents than 
graduates with diplomas. Table 8 presents chi-square results for tests on survey mode and 
phase with regard to respondents’ achievement. The distribution of respondents by 
achievement in Phase Two was not different from the distribution for nonrespondents. 
This suggests that late respondents in a mixed mode strategy tend to be akin to 
nonrespondents, at least on one variable, as has been found in mail surveys (Moore & 
Tamai, 2002). This suggests that the sequential modes have the potential to contribute to 
the reduction of nonresponse error by increasing the similarity between respondents and 
nonrespondents. It is interesting to note that graduates with certificates earned $1,782 
monthly, which was significantly less than the $2,187 earned monthly by graduates with 
diplomas, r(7 1 6) = 6.88, p < .0005). It has been found that nonrespondents tend to have 
lower socio-economic status than respondents (Goyder, 1987)). Another explanation is 
that certificate achievers, who are the College for one year, may not have developed as 
much of an association or involvement with the College as two year graduates. 

Mode effects 

Different types of collecting data from people may produce different results. One 
way of mitigating these concerns is through unimode questionnaire construction 
(Dillman, 2000). This was attempted in this research, as the survey instruments utilized 
for each of the modes were nearly identical. 

There were seven questions on each instrument, asked in the same way, 
attempting to understand the attitudes of graduates towards their experience at the 
College. Table 9 provides frequencies and means for all modes and phases. A one-way 
analysis of variance was calculated with each attitude question as a dependent variable 
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and Phase One survey mode as the independent variable. As Table 10 illustrates, there 
was a significant effect of Phase One mode on only one question (knowledgeable 
instructors), F( 2, 547) = 3.302,/? = .038. Bonferroni’s post hoc analyses revealed that 
mail respondents were significantly more likely to agree than respondents by web. Across 
all modes, the telephone survey produced the highest mean scores on all but one of the 
seven attitudinal questions. A one-way analysis of variance was also calculated with each 
attitude question as a dependent variable and all survey modes as the independent 
variable. The results of the analyses of variance and the Bonferroni post hoc tests are 
presented in Table 11. There was a significant effect with regard to three of the attitude 
questions. The post hoc analyses revealed that telephone respondents were significantly 
more likely to agree on knowledgeable instructors than web respondents, were 
significantly more likely to agree on appropriate technology than TDE respondents, and 
were significantly more likely to agree on reasonable costs than mail respondents. Given 
the number of possibilities on the Phase one comparisons, one effect does not seem to 
reflect a substantive difference effect through modes. There was some support for the 
notion that interviewer surveys tend to elicit more agreement from respondents. 

To clarify this investigation, a factor analysis was used and through principal 
component analysis extracted one component (only component with an eigenvalue 
greater than 1, see Table 12). The Cronbach alpha of the scale was .8163, (Scale 
statistics: M = 22.59, SD = 3.43, n cases = 895) indicating high reliability. The underlying 
dimension that was being measured was overall satisfaction with an academic program 
and it was expected that there would be just one component. A one-way analysis of 
variance with the summated scale score as the dependent variable and Phase One modes 
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as the independent variable produced a significant effect , F( 2, 478) = 3.21, p =.041. 
However, the post hoc analyses with Bonferroni revealed no significantly higher scores 
from any mode comparisons, although Mail contrasted with TDE was at p = .054. This 
suggested that any differences in response across modes in Phase One were at a minimal 
level. A one-way analysis was also calculated with all modes and the summated scale 
scores, F( 3, 891) = 3.03,/? = .029. Bonferroni’s post hoc analyses revealed that 
respondents by telephone were more likely to agree than TDE respondents. This analysis 
suggested that Phase One mode effects were minimal and there was very limited support 
for the notion that interviewer surveys tend to elicit more agreement from respondents, 
although this only applies to the TDE mode. It is interesting that the TDE mode is a 
mixed presentation. 

Discussion 

The reason for this research was to understand more fully the impacts of a mixed 
mode survey strategy in institutional research, particularly in relation to newer modes. In 
a time of declining rates and the requirement for reliable data for decision-making a 
mixed mode strategy may be appropriate. Improved unit response rates can reduce 
nonresponse error allowing for confidence in the findings of surveys, as long as item 
nonresponse is not critically compromised, representativeness is achieved and any mode 
effects are limited. 

There are several observations to make as a result of this study. First, there are a 
number of reasons for the trend of mixing modes (Dillman, 1999). The fact of declining 
response rates is just one. There has been an increase in the number of modes that are 
available for surveying such as fax, electronic mail, the Web, and voice activated or 
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touch-tone data entry. As well, people are more mobile today and live in changing 
arrangements and are likely best contacted by varying methods. It is challenging, if not 
impossible to reach all members of a survey population through one mode. The 
sophistication of current information and communication technology also allows for the 
management of complex surveys and makes using varying modes more efficient. There 
are numerous vendors and professional survey firms providing products and services 
using newer modes. 

Second, this research supports the value of mixed mode surveys, at least in 
sequence, for improving response rates. Simultaneous mixed modes did not increase 
response rates, as offering an opportunity to respond by different modes did not have a 
positive effect on rates. Both the TDE and Web-based modes had very low response 
rates. It may be that the letter accompanying the mail outs was overly complex in its 
descriptions of the procedures for accessing the modes. The TDE mode was described in 
detail on an additional sheet that may have led to confusion or may have been missed or 
misplaced. There was not a coverage issue as 98.7% of Canadian households in 1996 had 
a telephone (Dickinson & Sciadas, 1997). As well, the intent of a survey is to induce 
individuals to decide to respond. Perhaps the introduction of another decision point, that 
is, what mode to use, led to potential respondents delaying the key decision. During 2000, 
about 53% of Canadians indicated that they used the Internet at home, work or 
somewhere else. Internet users differ from nonusers, with nonusers more likely to be 
older, have less education and lower incomes (Dryburgh, 2001). This illustrates the 
potential coverage error for Web-based surveys. In this study web-based respondents had 
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a significantly higher reported monthly earnings than other mode respondents. Moreover, 
the continuing value of mail as a survey mode was demonstrated in this research. 

Third, TDE mode respondents exhibited significantly less care in completing the 
survey instrument, indicating that the mode appeared to lead to increased item 
nonresponse error. The novelty of this mode may have confused or discouraged 
respondents, as well as the fact that two methods were required for inputting responses by 
telephone key pad and by voice. Together with a very low unit response rate, these results 
suggest that TDE may not be an effective mode. 

Fourth, the mixed mode strategy produced a set of respondents similar to 
nonrespondents on two of the three comparison variables. There was a difference on 
achievement, with certificate holders more likely to be nonrespondents. However, there is 
some support for the notion that having a sequential mode may augment the 
representativeness of respondents. Phase Two respondents were not different from 
nonrespondents by achievement, while Phase One respondents did differ from 
nonrespondents and Phase Two respondents. This suggests that having a sequential mode 
may augment the representativeness of respondents. 

Fifth, the analysis of mode effects in Phase One indicates very minor mode 
differences on attitude questions, which suggest that mixed modes do not result in people 
responding differently depending on the mode. It also suggests that the attempt to have 
the survey instrument very similar across modes may have been successful in minimizing 
mode effects. There was very minor support for the idea that individuals tend to agree 
when interacting with other people. 
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There are limitations to this study and analysis of the impact of multiple modes 
for College surveys. It is a first level analysis with a focus on understanding the broad 
effects of a mixed mode strategy. The survey instrument was preexisting and was not 
designed specifically for a mixed mode assessment, although it was adapted for use in 
multiple mode forms. As well, the questions were determined by the graduates survey 
objectives and not this research. Response rates for web-based and TDE modes were very 
small, compromising the analysis. There were only limited demographic variables to 
compare respondents with nonrespondents. In addition, some of the variables were 
outcome variables that may determine a response set. Assessing mode effects is 
problematic. Just one of the other influencing variables is program taken; the program at 
the college can be quite different in content, technology used and style of instruction. 
Overall the scores on the attitude scale were high across all modes at an aggregate level 
while there are variations by program. This research was conducted at only one location 
and any generalizations are limited accordingly. 

There will be practical consequences of this study. Achieving high response rates 
allows for confidence in low nonresponse error and in using the results of surveys for 
decision-making. The graduate survey at this community college is one of the main 
sources of outcome measures and for assessments of program relevancy. It is used in the 
strategic planning process and plays a role in policy making. For 2000-2001 graduates a 
sequential mixed mode strategy will be pursued, using mail only followed by telephone. 
TDE will not be pursued in the near future due to high unit nonresponse and high item 
nonresponse. 
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Nonresponse in web-based surveys is a complex issue with many levels including 
the coverage, technology itself, rapid technological changes, the often mixed mode 
approach at solicitation, which occurred in this research, and the increasing spread of 
Internet access (Vehovar, Batageli, Manfreda & Zaletel, 2002). Although web surveys 
do have low unit response rates now, they seem to offer great potential. In this research 
the web mode did have a low response rate, however it appears that web respondents took 
care in completing the survey and provided the highest content on the key open-ended 
question. One of the issues with web surveys is a growing impatience with slow 
interactivity on some surveys (Crawford, Couper & Lewis, 2001). The survey form used 
in this research was not designed for high interactivity. The problems with noncoverage 
and nonresponse are not unique to web surveys. Moreover, the web has a great potential 
for audiovisual and interactive self-administered surveys (Couper, 2000). The 
deployment of this potential through appropriate principles and techniques will improve 
rates (Dillman, Tortora & Bowker, 1998). Although a Web-based mode will not be 
pursued at this time for this particular survey, in the future a further test will be 
undertaken. As survey success depends on the cooperation of potential respondents it is 
logical that respondents' propensity for various modes be considered by institutional 
researchers in designing surveys. 
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Significance tests on Achievement by Phase and Modes Respondents to Nonrespondents 
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Frequency distributions for all modes of attitude questions (No Opinions excluded) 
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One-way ANOVA and post hoc (Bonferroni) across Phase 1 modes 
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One-way AN OVA and post hoc (Bonferroni) across all modes 
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Factor analysis on seven attitude questions, total variance explained 
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Factor analysis , structure coefficients y Academic Program Satisfaction Scale 
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